{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ndo4ERqnwQOU"
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "MTKwbguKwT4R"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfNT-mlFwxVM"
   },
   "source": [
    "# 卷积变分自编码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0TD5ZrvEMbhZ"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/generative/cvae\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" />\n",
    "    在 tensorFlow.google.cn 上查看</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/zh-cn/tutorials/generative/cvae.ipynb\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" />\n",
    "    在 Google Colab 中运行</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/zh-cn/tutorials/generative/cvae.ipynb\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" />\n",
    "    在 GitHub 上查看源代码</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/zh-cn/tutorials/generative/cvae.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\" />下载 notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fKsm6LhC7TAw"
   },
   "source": [
    "Note: 我们的 TensorFlow 社区翻译了这些文档。因为社区翻译是尽力而为， 所以无法保证它们是最准确的，并且反映了最新的\n",
    "[官方英文文档](https://www.tensorflow.org/?hl=en)。如果您有改进此翻译的建议， 请提交 pull request 到\n",
    "[tensorflow/docs](https://github.com/tensorflow/docs) GitHub 仓库。要志愿地撰写或者审核译文，请加入\n",
    "[docs-zh-cn@tensorflow.org Google Group](https://groups.google.com/a/tensorflow.org/forum/#!forum/docs-zh-cn)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ITZuApL56Mny"
   },
   "source": [
    "![训练过程中输出的演变](https://tensorflow.google.cn/images/autoencoders/cvae.gif)\n",
    "\n",
    "本笔记演示了如何通过训练变分自编码器（[1](https://arxiv.org/abs/1312.6114), [2](https://arxiv.org/abs/1401.4082)）来生成手写数字图片。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P-JuIu2N_SQf"
   },
   "outputs": [],
   "source": [
    "# 用于生成 gif\n",
    "!pip install -q imageio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e1_Y75QXJS6h"
   },
   "source": [
    "## 导入 Tensorflow 与其他库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YfIk2es3hJEd"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "try:\n",
    "  # %tensorflow_version 仅应用于 Colab。\n",
    "  %tensorflow_version 2.x\n",
    "except Exception:\n",
    "  pass\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "import time\n",
    "import numpy as np\n",
    "import glob\n",
    "import matplotlib.pyplot as plt\n",
    "import PIL\n",
    "import imageio\n",
    "\n",
    "from IPython import display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iYn4MdZnKCey"
   },
   "source": [
    "## 加载 MNIST 数据集\n",
    "\n",
    "每个 MNIST 图片最初都是包含 784 个整数的向量，每个整数取值都在 0-255 之间，表示像素的强度。我们在模型中使用伯努利分布对每个像素进行建模，并对数据集进行静态二值化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a4fYMGxGhrna"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "(train_images, _), (test_images, _) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NFC2ghIdiZYE"
   },
   "outputs": [],
   "source": [
    "train_images = train_images.reshape(train_images.shape[0], 28, 28, 1).astype('float32')\n",
    "test_images = test_images.reshape(test_images.shape[0], 28, 28, 1).astype('float32')\n",
    "\n",
    "# 标准化图片到区间 [0., 1.] 内\n",
    "train_images /= 255.\n",
    "test_images /= 255.\n",
    "\n",
    "# 二值化\n",
    "train_images[train_images >= .5] = 1.\n",
    "train_images[train_images < .5] = 0.\n",
    "test_images[test_images >= .5] = 1.\n",
    "test_images[test_images < .5] = 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S4PIDhoDLbsZ"
   },
   "outputs": [],
   "source": [
    "TRAIN_BUF = 60000\n",
    "BATCH_SIZE = 100\n",
    "\n",
    "TEST_BUF = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PIGN6ouoQxt3"
   },
   "source": [
    "## 使用 *tf.data* 来将数据分批和打乱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-yKCCQOoJ7cn"
   },
   "outputs": [],
   "source": [
    "train_dataset = tf.data.Dataset.from_tensor_slices(train_images).shuffle(TRAIN_BUF).batch(BATCH_SIZE)\n",
    "test_dataset = tf.data.Dataset.from_tensor_slices(test_images).shuffle(TEST_BUF).batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "THY-sZMiQ4UV"
   },
   "source": [
    "## 通过 *tf.keras.Sequential* 连接生成网络与推理网络\n",
    "\n",
    "\n",
    "在我们的 VAE 示例中，我们将两个小型的 ConvNet 用于生成和推断网络。由于这些神经网络较小，我们使用 `tf.keras.Sequential` 来简化代码。在下面的描述中，令 $x$ 和 $z$ 分别表示观测值和潜在变量\n",
    "\n",
    "### 生成网络\n",
    "\n",
    "\n",
    "这里定义了生成模型，该模型将潜在编码作为输入，并输出用于观测条件分布的参数，即 $p(x|z)$。另外，我们对潜在变量使用单位高斯先验 $p(z)$。\n",
    "\n",
    "### 推理网络\n",
    "\n",
    "\n",
    "这里定义了近似后验分布 $q(z|x)$，该后验分布以观测值作为输入，并输出用于潜在表示的条件分布的一组参数。在本示例中，我们仅将此分布建模为对角高斯模型。在这种情况下，推断网络将输出因式分解的高斯均值和对数方差参数（为了数值稳定性使用对数方差而不是直接使用方差）。\n",
    "\n",
    "### 重参数化技巧\n",
    "\n",
    "在优化过程中，我们可以从 $q(z|x)$ 中采样，方法是首先从单位高斯采样，然后乘以标准偏差并加平均值。这样可以确保梯度能够通过样本传递到推理网络参数。\n",
    "\n",
    "### 网络架构\n",
    "\n",
    "对于推理网络，我们使用两个卷积层，后接一个全连接层。在生成网络中，我们通过使用全连接层，后接三个卷积转置层（在某些情况下也称为反卷积层）来镜像词体系结构。请注意，在训练 VAE 时避免使用批归一化（batch normalization）是一种常见的做法，因为使用小批量处理会导致额外的随机性，从而加剧随机抽样的不稳定性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VGLbvBEmjK0a"
   },
   "outputs": [],
   "source": [
    "class CVAE(tf.keras.Model):\n",
    "  def __init__(self, latent_dim):\n",
    "    super(CVAE, self).__init__()\n",
    "    self.latent_dim = latent_dim\n",
    "    self.inference_net = tf.keras.Sequential(\n",
    "      [\n",
    "          tf.keras.layers.InputLayer(input_shape=(28, 28, 1)),\n",
    "          tf.keras.layers.Conv2D(\n",
    "              filters=32, kernel_size=3, strides=(2, 2), activation='relu'),\n",
    "          tf.keras.layers.Conv2D(\n",
    "              filters=64, kernel_size=3, strides=(2, 2), activation='relu'),\n",
    "          tf.keras.layers.Flatten(),\n",
    "          # No activation\n",
    "          tf.keras.layers.Dense(latent_dim + latent_dim),\n",
    "      ]\n",
    "    )\n",
    "\n",
    "    self.generative_net = tf.keras.Sequential(\n",
    "        [\n",
    "          tf.keras.layers.InputLayer(input_shape=(latent_dim,)),\n",
    "          tf.keras.layers.Dense(units=7*7*32, activation=tf.nn.relu),\n",
    "          tf.keras.layers.Reshape(target_shape=(7, 7, 32)),\n",
    "          tf.keras.layers.Conv2DTranspose(\n",
    "              filters=64,\n",
    "              kernel_size=3,\n",
    "              strides=(2, 2),\n",
    "              padding=\"SAME\",\n",
    "              activation='relu'),\n",
    "          tf.keras.layers.Conv2DTranspose(\n",
    "              filters=32,\n",
    "              kernel_size=3,\n",
    "              strides=(2, 2),\n",
    "              padding=\"SAME\",\n",
    "              activation='relu'),\n",
    "          # No activation\n",
    "          tf.keras.layers.Conv2DTranspose(\n",
    "              filters=1, kernel_size=3, strides=(1, 1), padding=\"SAME\"),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "  @tf.function\n",
    "  def sample(self, eps=None):\n",
    "    if eps is None:\n",
    "      eps = tf.random.normal(shape=(100, self.latent_dim))\n",
    "    return self.decode(eps, apply_sigmoid=True)\n",
    "\n",
    "  def encode(self, x):\n",
    "    mean, logvar = tf.split(self.inference_net(x), num_or_size_splits=2, axis=1)\n",
    "    return mean, logvar\n",
    "\n",
    "  def reparameterize(self, mean, logvar):\n",
    "    eps = tf.random.normal(shape=mean.shape)\n",
    "    return eps * tf.exp(logvar * .5) + mean\n",
    "\n",
    "  def decode(self, z, apply_sigmoid=False):\n",
    "    logits = self.generative_net(z)\n",
    "    if apply_sigmoid:\n",
    "      probs = tf.sigmoid(logits)\n",
    "      return probs\n",
    "\n",
    "    return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0FMYgY_mPfTi"
   },
   "source": [
    "## 定义损失函数和优化器\n",
    "\n",
    "VAE 通过最大化边际对数似然的证据下界（ELBO）进行训练：\n",
    "\n",
    "$$\\log p(x) \\ge \\text{ELBO} = \\mathbb{E}_{q(z|x)}\\left[\\log \\frac{p(x, z)}{q(z|x)}\\right].$$\n",
    "\n",
    "实际上，我们优化了此期望的单样本蒙卡特罗估计：\n",
    "\n",
    "$$\\log p(x| z) + \\log p(z) - \\log q(z|x),$$\n",
    "其中 $z$ 从 $q(z|x)$ 中采样。\n",
    "\n",
    "**注意**：我们也可以分析性地计算 KL 项，但简单起见，这里我们将所有三个项合并到蒙卡特罗估计器中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iWCn_PVdEJZ7"
   },
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(1e-4)\n",
    "\n",
    "def log_normal_pdf(sample, mean, logvar, raxis=1):\n",
    "  log2pi = tf.math.log(2. * np.pi)\n",
    "  return tf.reduce_sum(\n",
    "      -.5 * ((sample - mean) ** 2. * tf.exp(-logvar) + logvar + log2pi),\n",
    "      axis=raxis)\n",
    "\n",
    "@tf.function\n",
    "def compute_loss(model, x):\n",
    "  mean, logvar = model.encode(x)\n",
    "  z = model.reparameterize(mean, logvar)\n",
    "  x_logit = model.decode(z)\n",
    "\n",
    "  cross_ent = tf.nn.sigmoid_cross_entropy_with_logits(logits=x_logit, labels=x)\n",
    "  logpx_z = -tf.reduce_sum(cross_ent, axis=[1, 2, 3])\n",
    "  logpz = log_normal_pdf(z, 0., 0.)\n",
    "  logqz_x = log_normal_pdf(z, mean, logvar)\n",
    "  return -tf.reduce_mean(logpx_z + logpz - logqz_x)\n",
    "\n",
    "@tf.function\n",
    "def compute_apply_gradients(model, x, optimizer):\n",
    "  with tf.GradientTape() as tape:\n",
    "    loss = compute_loss(model, x)\n",
    "  gradients = tape.gradient(loss, model.trainable_variables)\n",
    "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rw1fkAczTQYh"
   },
   "source": [
    "## 训练\n",
    "\n",
    "* 我们从迭代数据集开始\n",
    "* 在每次迭代期间，我们将图像传递给编码器，以获得近似后验 $q(z|x)$ 的一组均值和对数方差参数\n",
    "* 然后，我们应用 *重参数化技巧* 从 $q(z|x)$ 中采样\n",
    "* 最后，我们将重新参数化的样本传递给解码器，以获取生成分布 $p(x|z)$ 的 logit\n",
    "* **注意：**由于我们使用的是由 keras 加载的数据集，其中训练集中有 6 万个数据点，测试集中有 1 万个数据点，因此我们在测试集上的最终 ELBO 略高于对 Larochelle 版 MNIST 使用动态二值化的文献中的报告结果。\n",
    "\n",
    "## 生成图片\n",
    "\n",
    "* 进行训练后，可以生成一些图片了\n",
    "* 我们首先从单位高斯先验分布 $p(z)$ 中采样一组潜在向量\n",
    "* 随后生成器将潜在样本 $z$ 转换为观测值的 logit，得到分布 $p(x|z)$\n",
    "* 这里我们画出伯努利分布的概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NS2GWywBbAWo"
   },
   "outputs": [],
   "source": [
    "epochs = 100\n",
    "latent_dim = 50\n",
    "num_examples_to_generate = 16\n",
    "\n",
    "# 保持随机向量恒定以进行生成（预测），以便更易于看到改进。\n",
    "random_vector_for_generation = tf.random.normal(\n",
    "    shape=[num_examples_to_generate, latent_dim])\n",
    "model = CVAE(latent_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RmdVsmvhPxyy"
   },
   "outputs": [],
   "source": [
    "def generate_and_save_images(model, epoch, test_input):\n",
    "  predictions = model.sample(test_input)\n",
    "  fig = plt.figure(figsize=(4,4))\n",
    "\n",
    "  for i in range(predictions.shape[0]):\n",
    "      plt.subplot(4, 4, i+1)\n",
    "      plt.imshow(predictions[i, :, :, 0], cmap='gray')\n",
    "      plt.axis('off')\n",
    "\n",
    "  # tight_layout 最小化两个子图之间的重叠\n",
    "  plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2M7LmLtGEMQJ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100, Test set ELBO: -78.06732940673828, time elapse for current epoch 1.886899471282959\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOwAAADnCAYAAAAdFLrXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2daWBU5fm3r5lMEmQXhbApEFQWUaOt4F/BDRQUV1SkaFFLRVurttUKdYfaauvSulS0KC1WRRZRobgBokBdEUWh4gYoQiSyyxIgybwfzvt7ziQZYpbZTrivL4HJJDnPnPM8937foWg0imEYwSCc7gswDKP62IY1jABhG9YwAoRtWMMIELZhDSNARKr6ZigUCrQLORqNhqrzvr1lnbD3rLW+rtMkrGEECNuwhhEgbMMaRoCwDWsYAcI2rGEECNuwhhEgqgzrGOklJyeH0tJSAPc108nKygIgGo1SVlaW5qtJPFpfo0aNAOjZsyc9evQAYL/99gNg6dKlAEybNo1du3Yl9O+bhDWMAGESNsFEIt5H2qRJE7Zs2QJUTzpGIhGOPfZYAO644w4AOnTowHnnnQfA+++/D3iSKxPZf//9AVixYgUA27Zt4/jjjwfgs88+S9t1/RChkJef0LhxYwAGDx7MddddB0CnTp0AyM3NrfT+6lBSUsJJJ50EwIIFCxJyvbXesFINTjrpJK6//noA2rVrB3g37Y9//CMA7733HsAPqkcNGzYE4L777nO/F2Dy5MmMGTMG8D4AyNyHFryNCvDkk0/SokULALfpioqK3Oegh6BLly4AnHLKKVx22WWA/6BkZ2e792XymgGefvppwH/wGzduzPTp0wEoKCgAoLi4OD0XV4GcnBzAuy+PPvoo4N+3H0KHr1TdNWvWsGjRIvf7AMJhT3GNRCLMmjULgFatWgHw/fff1+naTSU2jAARqurkriofs0OHDoCnqkmSxFMXdKp+/PHHAHz99dds2rQJ8NWo/v3706BBgz1ex5IlSwAYMmQIAJ988km1HBrpyDuVRFy6dCn5+fkAvPXWWwD87W9/48svvwTg22+/BXCfRTQadaf8hAkTAOjRowfdunUDYOvWrXv8m+nOJQ6FQhQWFgKQl5cHeBrV9u3bATj66KMBWLZsWZ3/Vl3uqZ7Pa665BoB7773XaYrxkEa3cOFCAC6++GKn8sc+fzKD5syZA+BMgVg6d+4MwPLly6tz+ZZLbBj1gVrbsDqtcnJyqjTEJTl1yuprTTjggAMAOPHEEwFYuXKlO70zzbaTjdOkSRN3bc8//zwAM2fOZMeOHUD865Y2Mm/ePMCTSJli91VFNBqt5FjbtGmTs/OkSaUbSUI9R2VlZZUkbDQa5aKLLgJg0qRJ7n1VIa2qZ8+ee3zPxo0ba3XNFan1hl27di0A3333XbUN9tqybds2wFe32rRpw+rVqwH/Ic+0jTtv3jymTZsGVP/GC63p/vvvd2pZpqMDVNe7YMECZyrJpEk3u3fvBuCRRx4B4LDDDnMOPh0uRx55JJ9++ilQ/Wfq3HPPBYhr1ulvJurgNZXYMAJErSWsToxRo0bx2GOPAdC0aVP3fZ20cq6sW7cOgBYtWrjwT1UGv9i9ezcvvPAC4Dmb9JrCBzoZMyUTSNcxevRod73Vlaz6PBQmkBYRBDZs2AD4zsgDDzzQqfabN29O23XFY+7cuQBcffXVnHzyyQDcc889gK851oSxY8fu8XtyRO3cubPGvzceJmENI0DUWsJKv3/55Ze5/fbbAbjqqqsA+PLLL112y4svvlju50455RSGDh0K+LmXWVlZe3RcrVu3zoU0srOzAWjWrJmzmxUWyRQJK8m/Zs2aWl+TsmIyZU3V4aOPPgL8JImOHTty4403ApnnX5BW9tJLL7nEhtrmPbdr187d83hIwtYkQ6oqTMIaRoCocy7x1q1bGTduHODnuzZv3pyuXbsCuPxYBY6Li4t59913AT8FLz8/v1y+Jvin4LZt29z7JJF3797N+vXrgczJU9Up279/f6CyZvFDhEKhwHiE4yF7W6GTSCTC/Pnz03lJ1aK2n7kkZlXJIKWlpe451vO9Y8eOOmkcdd6wsWVUipeOGjWKgw8+GPDVWFFSUuJUXDkjysrKnPqnXMvPP/8c8DJDlGd86KGHAl7st2XLloDv1JJKlmq0vn/84x+AnxO9bNky53SqjmqbaWpjTWnTpg3g59FGo1EXjqtPaH3KRounDut+b9682Tnh2rZtC3iZfnUpuTOV2DACRJ0lbCgUok+fPgBONd5nn332+P6cnBwXUNdX8CWMpO/UqVOB8hJ2wIABgGfoy7kxbNgwAFcxlErC4TAjRowAfNVfWsPvf/97FzSXFvDhhx/y+uuvA17lDtTe2ZFpSKMSZWVlgdcalAjRrFkzwDN3HnzwQaB8CFNIcr722muA9+zKgShTITs72ySsYewt1FnCdu7cmYkTJwJVS9YfQka8pNE777wDeKeUgs76OnjwYCd1Y6V0qikoKKB9+/YA3HTTTYCfhldaWuo0j5EjRwLw61//2tl1quTJtKSC2iIbVlI1FAq5exokSZuVlUXv3r0BXGrpvvvuC+w5NCPH1QMPPADA3//+d/e9gw46CIBevXoBcMQRR7g6YeWV14Q6J/+PGDGiknoQjUbdg6hYnMqKBgwYwPDhw4HyRcNSDWfOnAn4HuedO3e6v/Xf//4XgO7du3PGGWcAOBUzlUhV6t69u1ORVF4W+3Bq86pI+rnnnuOss84C/CKI2bNnp+aik0ish1v3MTYbra5F26mkRYsWTgBVVxio5G7KlCmAfwi3bduWvn37An7Bwbp161ymVW02rKnEhhEgai1h5d4+6qij3Omqio2zzjqLN954A6isDr3yyiuMHj0a8HODW7du7TKWxo8fD1DOMJeElcrdpUsX9/10SCiFcmbPnu1U+KqQ1LnkkktcmZXU5PoiYaVNqBVKJBLh9NNPB/xqpSBw2mmn0bp162q/PxqNuudeYcfDDz8cgBNOOIEf//jHgF9p1rBhwzrlFZuENYwAUWcb9pNPPqF79+4Azp574403qnQ0yPGiCp68vDwnhXRKSYpFo1Hn0LjyyisB6N27t6uqUHgkFcg1rxOyprbZtm3b3DolieoLzz33HODftz59+rg60SBIWFVK3X777TXK+925c6dLlDjzzDMB3H5o06aN0wr1+9euXWsS1jD2FmotYXWqzJkzh4EDBwJ+i4zc3NwqK+yVEywpEwqFXBuRyZMnA77tm5OT46SSTqnt27fzxBNPALXPBa0poVDInZa17R7QoEEDZ6urhrg+EI1GXbd7RQNOPfVUl08ehPCOQjkHHnhgpe/punfu3Om0wjfffBPwulcojVY/qyZvbdu2dc+snuFFixbVKXGizuV1M2fOdO57Zf08++yz3H333YC/MG2s5s2bO9VZGzcWZZXEosXKuF+0aFHKH/h99tnHFTCo4ECZTNWld+/eLjQl51p9IBqNutCG1L2gFDMoRPfwww8D5ZsqaC333nsvAE899ZQrOlFIMhKJOOGlg1xOq0gk4vaJvjd58uQ6HVymEhtGgKhzptPu3bt58sknAZgxYwbgjZr4xS9+AeBc+0osOOOMM1zebXVaxAB88803ADz00EOAp05KtUw2Uue6d+/Oz372M8A/javbXEyaxCWXXMJf//pXoHZB80xGYT5VUWVnZ7t7nsnIjJP2BL72+OGHHwL+c71r1y53L6+++mrA05okpVVCp8kCRUVFLoynZ7iwsNA997XRQEzCGkaASOgwLKVkXXvttS5dsWPHjoA/Kyc/P79SjWw8Yo30W2+9FfCTDGpqOyaCdevW0bx5cwBuuOEGAH73u9+5sFJsDi14p61Ob7n7lyxZ4mYNZbIDpjbIbtN8maysLJdmmslrlWagEF0oFHKvKd/7kksuAbzaV+Wwx/Yg1vs1IUDSdOHChaxcuRLw+xLr/7Wl1qM6aorUgIKCArfxtAFi0UbVUKzRo0c7Z1NNS9ESOaojEolwxBFHAHDFFVcAXvxYN6BiEX+nTp3cdd91112Ad+AkqnteLOke1ZGbm+tyupXZs3z5cg477DCAhM5ITfT4FTk5Zeb069fPJftLZdWYmWXLljk1WQ0TFi9e7ARVxWFtsZtfajJUHqgVDxvVYRj1gJRJ2FgUf1Vopn///k4Cy4GlEFFdTudEn8a6RoWxTjjhBE455RQAvvrqK8A3Cz766CMXm1RmV7JUw3RL2I4dO7o+XdI0OnfunJQWMckacCYzLSsryzmRJDHlIExlF0uTsIZRD0iLhI35/YBnR6hu8NVXXwUSU0OZjnGT6SDdEjYUCrlEAiUIJNJujWVvv6cmYQ0jQKRVwsYib1oim5Lt7adxPPaWtdbXdSY0DlsX6kv3QMNIJqYSG0aAqFIlNgwjszAJaxgBwjasYQQI27CGESBswxpGgLANaxgBwjasYQQI27CGESBswxpGgKgyNbG+5mNWZG9ZJ+w9a62v6zQJaxgBwjasYQQI27CGESBswxpGgLANaxgBImMK2GuKOlQIK4A39gZMwhpGgEiLhFW3RH3NyspyElO9YCUxQ6GQ6xmr7umRSMQNHtqyZQvgd+szSZteqjsLVtPsRRBGU9aF2Ge+Ls9o0jasLlDjOI466igAhg4dSocOHQC/Iff333/v2po+9dRTgD+nJDc31w1SViPnrVu3ur+jDZvJnTP0WUBmX2dd0BqHDBkCeJPb1Fy8YsvTNm3auA26evXqFF5lZSQoJADy8/M5/PDDAfi///s/wJ8L1bJlSzfXSYOdx44d62b+fvvtt+V+Z4sWLdxsKTVV//jjj12z+Vpdb61/0jCMlJOUNqe5ubn85S9/AeDKK68EfBUo9u9JNdi6das7gZYvXw7A1KlTAXjxxRfd1OtYdVm/R6d3vDEK6Uxjy83N5dJLLwVg+PDhgDfR+4EHHgBg7ty5AJUGKcVSXWmcCamJMlveeust/R3uuOMOwJ9y3qVLFwBeeuklFi1aBNR8EmGi76nmvWoY1tlnn11pbrEkZkVH5/+/HmeOSepqvdnZ2W6EyzXXXAN4n0911mypiYZRD0iKDbvvvvtyzjnnAP7JKwmyZs0aNyRKo/u2bt3q5ovqFPv6668BWL9+faVhUtFoNCmNxxOB5uLOmTPH2e261tLSUnr06AH4Nvpnn30GwIQJE/jiiy8A397ZtWtXYGxe3Qc5BtevX8/+++8P4CbXa10PPPBAWmb8xkPjPw855BDA0wSlrWnOq57Xxo0b061bNwA3khJwGqA+A/lctm/fzrXXXgvAggULyr2ntiRlwxYVFbkhxlIFXnnlFcCbkaq5qbGDc3/9618D0LZtWwD3wSxdurSSBzh20bEOHUifU6dNmzYAvP3224A3J7aixzQUCtGoUSPAd8JJ/WrQoIF7iCvOGQ0CWqse3sLCQoYNGwbAgQceCMDgwYMB2LBhQxquMD5yYP7iF78AYOLEic5Mue222wDv8AVvip3WqQ17/PHHM2jQIPdv8M2022+/nfnz5wOJEyymEhtGgEiKhC0rK3NTq3/7298CvrQoLi52DihN6L7rrrv40Y9+5L4f+3X+/PnOmNdrpaWllaRPuqSRwlaLFy8GPNe/rkenqiRmSUmJ0y40hf7+++8HYNGiRU7CBkmyCpk++fn5gDcfVubBxIkTAd8hlYm88847gKe+a+bv2rVrgfLPndi0aRPgmQCnnXYa4IeGZs2aBcC4ceMSbrKZhDWMAJG0xAlJCSVE6PRp27YtN9xwA4ALe+Tm5jrbQKfYd999B3iTzTUBO9aGzQQpFA6HueqqqwAvSA5+mOKrr75yjgY51IqKipyNq1NYiQO7d+/OiDXVFtnmsf/XPRw1ahSQeQ7CWPTZv/766yxbtgyAnj17AjhH4ccff+wcStIaBg4c6J5d2cPXX389kJzsLZOwhhEgkp5LrJNLJ/CLL77oAuixAWqdvpJQkjybN292XrdMk0ChUIhzzz0X8K//wQcfdF8VMujVqxfgfQZr1qwBfLso9ndl2vpqQsOGDQHf879z507GjRsH+OmjQSAajVJYWAj44ZwLLrgA8CIXur9KuIjl+eefB/zkn2SQsuR/xeK6detWKRSzdetWZ8TLKaMwSVZWVsY+yNnZ2e7w0eYcO3Ys4IU1lCutw6pNmzYUFRUB/oaV2hQKhaqdOJ+JHHzwwYC/Yb/44gsmT54MBHM9gMtS0uYcPXq0c67FItX/lltuAZKr+ptKbBgBIukSVlJj6NChgHf6bNy4EfCdEbNnz6ZZs2YAPPbYYwCcf/75ADz99NPOeZNpVCwRA1wlUklJCUceeSSAC1l9/vnnTl1S1o9O4z1lbwVBOoVCIVq1agX4175+/fq0V+LUFTlAR4wYARBXugIuLzoVJYImYQ0jQKRMwiqBomHDhi5pQHZf7Psef/xxAB566CEAjjvuuIyVsLF1nrJT//Of/wCeo0Xf/+STTwAYP368q5mMtV31tWKVSDQadad8bB517PczgVAo5BxL0p4KCwsDX5SuqpsmTZrs8T2lpaVO8iq0J80iGfcn6RtWD6Hij8XFxXEXoteUMST1UFlOmciuXbtclovyTXWTc3JynHqoTXfooYe6B1uv6WZHIhH3s/K4bt++3eXd6nPI1GyoAQMGAL6DZtasWRl3jTVFMdd4SNhs3LjR5b/3798fgCVLlgDJuUemEhtGgEhKATv4ZVb6qnDND7m8VSGhr61ataq1lE1lAbsyuaT69+vXjyOOOALAZWp98MEHPPfcc4CvOiq807VrVw444ADAL+uaO3euU6djzYeKpLuAvWnTpk6q6Ot5553n1p1IUnlPlSeuexX772nTpgHQp08f1wZGWpMy2+oSf7YCdsOoByRUwsY2XpObX9JRttie/p5+VkFoJR3IrqsN6WwREw6HnU3z6KOPAl6Fh5xp77//PuCf2KWlpc5Jo8+gpKSkWkH4dEvYgoICXnvtNQBGjhwJ4LKcEk0q76meyXnz5gHw3nvv8cQTTwD+8zxgwADXDknVSccccwyAa0JXG0zCGkY9IKFeYrXZGDdunDuBbr75ZsCvH4zXLA3gX//6F+BX8l900UWJvLSUU1ZWxquvvgrAjBkzAMjLy3PVS9IgVHNZVFRUqR1opiNv9mWXXebW88EHH6TzkhKKtMHLLrsM8NqYyhej702YMMG1Q1X12U9/+lOgbhJ2TyRkw8qxNGnSJMArR5IqfNBBBwGwcuVKwAuFSPWTo2bu3LkuQV7J8c8880wiLi2t6HDS53L55Ze7HrerVq0C/DBWkDarQm4qOxs8eLB7TY0L6hOKq8YLSe7atcv1HxMFBQVJuxZTiQ0jQCREwirDo2vXroB3AstgVzM2fe+0006je/fugB9kjx1foIZd9QlVHh133HFOdfzzn/8MUKcu8OlCGpXMlhYtWrjSsqrCT0FDST96Jr/44otKJl12dja//OUvy70m8y8ZmIQ1jACREAmr1hiyw3Jzc11PWhngIl7FQ0lJiXM27ckpFURk1/3kJz8BvEoe2e+y0YOUvietSRrViSeeCHjOmKA7CeMhH4sSWtasWeOcTsodnzp1qnvWxZNPPpm0a0rIhlWpmG7aM8884/IwVYIWW7SuTamY1vDhwwP14FYX9QQ644wzAG8Da3BSEEvP1BFy+vTpAHTq1AnwhkXVJ1VYKAdADtHevXtz6KGHAn7utEwc8HsyyzxIBqYSG0aASEqm0/7770/fvn0BP/tDObNvv/22G0mR7C566cx0atSokctikhlQWlrqMsAS2f0+FZlO4XDYTW/o168f4PcuOuigg1KmIaXynkp6qtVNv3794mqMcjJ17twZSMy9tUwnw6gHJDTTSafsd999Vy8SH+rCTTfdVMnBdvbZZ2fUXJmakJWV5dajpBhVI9VH/wP4FWbSLI466ijniJJUvfPOO3n66aeB8oPGk4VJWMMIEEmrh80E0mnD3nbbbVxxxRWAP9D5pZdeSvSfAVJXrSP7Ld4UwVSRznuaSva0TtuwJGedkUjEqYrJji2nu7wuleztG9ZUYsMIEFVKWMMwMguTsIYRIGzDGkaAsA1rGAHCNqxhBAjbsIYRIGzDGkaAsA1rGAHCNqxhBAjbsIYRIKosr6uv+ZgV2VvWCXvPWuvrOk3CGkaAsA1rGAHCNqxhBAjbsIYRIGzDGkaASGgTNmPvQpMNGjduTPPmzQF/GHW8SW9G3TEJaxgBwiRsglBjaUmdRo0a0b59ewAOO+wwAJo1a8ann34KwFtvvQUEay6sUKvPQYMGAXDvvffSpEkTABYtWgR484Q067e+o3uuqX7SLHbt2lVJywiFQu79amqneUvV6f1V5w0biUTcWL5kzFeJ7bCuf0ej0bSqW+rPu99++3HMMccA/tCvY489FoDmzZu7zyW2GZtu1ueffw74c3e++uqrFF197dED9vvf/x6AUaNGAf4GBr9Xca9evfjPf/4DwO7du3/wd8fe5yCo0rqPBxxwANdffz3g9S0GbzgYwIwZM3jvvfcA2LFjB+DNodLPfv/990DNmvSZSmwYAaLWElZzR+bOnesG3h599NEAlUbI1wT9rgULFgDeyTRw4EAAVq1aBaRGjayotnTs2JGrr74agPPPPx/wZghJilakrKzMSRapPJs2bXLOGQ24njhxIuBNRktHn9/qEgqFuPbaawG49dZb3WtCUnHt2rWA1wW/qvVISxkxYoR7bdKkSYA/WSBZ6LpDoZC7f7rWqrS3WHVWJsAJJ5xA27ZtAdzcJA3wXrhwoftbkqbbtm1zElXPRU3uu0lYwwgQNZawOjGkrx955JHupLjgggsAePjhh53OXh3y8vL47LPPAH/andiwYYOT2KmQrJoJ2rFjRwDOOussAG644QaaNWtW7r3hcNjNxpWt8u677wKeTapZKz/+8Y8BTys55ZRTAGjXrh0A3bp1A7zZNZksYXv16sXdd98NlJesQp/DddddB3ga0p5ss3A47GbMas5qSUmJm4yQbAkbex2S9FpTw4YN3fd1/XqWY300mrvz+uuvu+HOmh37ySefADB+/HgX5krUva31ht28ebO7uPHjxwPw6KOPAv5oyR9CQ5+/+uqrco6L2N/Rpk2blHlSQ6EQrVu3BmDIkCGAr7I1a9bMqbgaynzbbbe5jVrRsRKrbslzevnll5dznOl94B0U1XHOpBrdlzlz5sTdqACFhYVceumlAMybNw+o2gHZs2dP+vfvX+61SCTizKyKn1Gi0e8tKSlxm1Kq7j777OMmqst8WbJkifs5vV/X2LVrV2688cZy7//Nb34DQFFRUcLXYCqxYQSIGktYifaPP/4YgB/96Ee1Mp7BC4tA+bCAJKtU41RKnWg06lSYKVOmAL4TZcWKFbz99tsAbNmyBah6vbGnsRxpAwcOdOqTflaSSCd8pvHggw8C5VVFIVVx+PDhzJ8/H/DXE0+yHHTQQYCnRsaT1hXjmKlAf0v3o7S01Dka9VXPZ3FxsbtPcjSNGzfOxdu//PJLAGbNmlXudyeSzHxKDMOIS63DOjqR6mJfnnDCCe7fOo3kEk+XPSdnghwHS5curfPvlAayYcOGShrE8uXLgeQkndQFOWNkm8YizUF2fqyDKTZkIgnVo0cPAGbOnAlQyV8BnvSS1pYOYiVtixYtALjmmmsA6NKlC+DZpHo+9Fr79u3dz5566qmAf7+TgUlYwwgQac0lPumkk9y/X375ZcAfRZ8udFomwv7Q71De8Hnnnee+9/Of/xzwUxIzTcJ27twZ8CUt+Ov53e9+B8Crr74KeFpWrJcVoGXLlm6QtdYtn0U8Jk6cmBFe8i1btrh8b9mk/fr1AyA/P99pENIootGoe2ZTkV6a1oHOSg7Py8vj4IMPBnwVMRFkWsOuUCjkHCvTpk0DfBX8wgsvrHWsLhlN2MaOHQvAlVde6V5TfFQqrv4fjUbdhpU6OWjQIG6//XYA9t13X8B34sQ62OS4Ovjgg1m9evUPXlc67qnCPCeddJI7aKUSt2rVymXlKVSVoMPemrAZRtBJi0qsALkCzdu3b6ewsDAdl5JSYiVRnz59gPhZQ5mAsrNikUSN51SJrUgCWLZsmUsqKSgoAHxJlZOT49Z9//33A36FSyaidT/77LMuMUTaw89+9rNyecjJxiSsYQSItEhY1ZDqlF23bl2NagKDjKSTcpaVvphpucSyy2IlrTQj5Vcr93fLli1uXbLJly5dysMPPwzA8ccfD/j52X379nWahhIzMmnte6KsrMwl1ujzGTFihLPRU0HKN2wkEuHpp58GfA/kjBkz9poNK7VJGzVTM5wqejyj0aiLHd9yyy2Apw4CTJgwwWX3FBUVAV4xwIoVKwC/FE05uWVlZc4BpZz0IBStxyLPcTgcdqp+KsjMp8UwjLikTMLqRJ09ezYtW7YE/DzdUaNG7TUSVp9DRWdTJBLJiDgkeFJDrW4k+bZt28bKlSsBXzVWO5g//OEPXHHFFYCXWwteVpdybBWukTQNh8OBdTLKlFFYB1Lbl8skrGEEiKRLWJ1If/3rXwHPAaFTu2/fvoDvqNgbOPfcc4HKNZ/JzD+tKbEOMBWmv/TSS0ydOhXwq4+GDRsGeNJGdtydd94JeLasmhK8+eabABx++OGA16hu9uzZQOaGteLRsGFDevXqBeASfcC321OBSVjDCBBJk7DygkrXj82jVTggEZUwQSI7O5t//vOf5V5TUD6TbPiysjLn0c3Pzwdg9OjRzoaVZ1tS8pFHHuGQQw4B4tu8krSxiTL6/Zlit8cjtsc0QKdOnTjzzDMByjVSu++++1J2TUnbsFqQHA0qqSopKWHo0KHJ+rMpIbZpeHU2mta+ePFi92892L/97W+BzIpDlpWVuS5/Un9PP/10PvroI8DfxPrezp07+eabbwC/PLJx48Yum0vOJxV9h8Nht4lFKBTKiNBOJBJxTtHTTz8d8J/hBg0aOKeh4rEbN27ktddeS9n1mUpsGAEi6U4nqcZyPq1atcq1WAkqUpF++ctfutzSxYsXA76K17hxY9fVX83pYtusKDFBZYWZImGEqmh030aOHOlK55TwEjtiQuuWBpGbm+s0kU6dOgG+c/WqRY4AAAybSURBVHHZsmWuJE3v3717d1rWX7Fb4p/+9CfX2VL3+fnnnwc8rSgvLw/wK81mzpxZ7aaDicAkrGEEiKRLWJ2aKtC++eabM0qS1AZJnSFDhjBmzBig/AAk8E5s2TuxSMqovcrGjRsB76TXaZ8J9uz7778P+FU07dq1q2R/a607d+50zjMVfy9atMglR8jOl1TasmWL6zWd7rBOxSqjKVOmOGeaGixoTa1bt3ZrUNKPcopTRdIL2NUpT6rfySefXKdRHjUh2cXOTZs2dYXo6k8Vb5OK7du3u4dAGyIR3uFkFLDLQypVsUWLFk4V1uGrRuk7duyo1DnzB56rcr2f9HPVOciTfU/D4bBbu2LL6lWdl5fn1q6DbOPGjU69l1mQiK4lVsBuGPWApKvEOn3U/6e4uDjpnd1TxZYtW1zVhtRFDQQbPHiwG+0ht/+UKVMCk9UlSSkpqq+JIN3jQqsiVtIrdCN1f+nSpZXa3JSWlrrPqrZaQ00wCWsYASLpNqwkj4LQy5cvdz1/k13lkGlN2JJFMmzYTCVT7ml1nGVmwxrGXk7SbVh5QT///HPA8zrK81jRq2YYQSFdz2zSVWKpDrGTymObMCeTTFGfko2pxJWpr+s0ldgwAkSVEtYwjMzCJKxhBAjbsIYRIGzDGkaAsA1rGAHCNqxhBAjbsIYRIGzDGkaAsA1rGAHCNqxhBIgqk//raz5mRfaWdcLes9b6uk6TsIYRIGzDGkaAsA1rGAEiZQOdDaO+k5OTA/h9q5Mx4cIkrGEEiLRKWHWh6Nu3L+eccw4Aq1evBuCpp54CoLCw0DVrs9rdzEf3NCcnxzUXz+SRknWldevWPPTQQwAMHDgQ8GcO9enThw8//DChf88krGEEiLRIWE0F05ySpk2buu9Jit58880ALFmyxA3Mfe655wBvVEQmSNtQKOQaSotMGsycKsLhMJdccgmAkzY5OTnus3jzzTcBOOuss4DENiVPF7JTL7roIjdPVkijKCkpSXjT/KQ3YatIgwYN3LDgeHNopE5IDQ6Hw66z+uOPPw7AmDFj3O/QQxGvsVsig+yhUIhjjz0WgGeffRaAli1bug1bcajS1KlTueyyywCSPo4wXYkTsZPYNTMoFt03fdVMpRNPPJFVq1bV6m+mO3FCay4oKAA8YaPJAGPHjgX84VlXXXWVm8VTUyxxwjDqASmTsFINNm3aVE4FBm/62U033QTAG2+8AUCHDh0AuOWWW+jatSsARUVFAAwaNIj//e9/gC+J441oTORp3L59e9dbWepQTdm+fTtPPPEEAKNHjwb8+S11UaXTJWElUa688spK31u1apUbWn3IIYcAuFlDS5cudTOJNG6zuqRTwoZCIVq0aAF4A7sBNmzY4F6bM2cOAPfccw/gDfKurSpsEtYw6gEpczrdddddQHkH0zfffANAjx49XJBZkvizzz4DoFevXnTs2BHwJeyXX37pbINUOZ9atWoVV7JKMq5fvx4oP7lMzojYWauSRvoqB0yHDh3YsGFDEleQOPbbbz8Arrjiikrfk4Y0fPhwt7a8vDwALrjgAsBzPt1xxx1AeU0jExyJVRGJRNy91LMbCoUYNGgQ4M+TXb58OZCcZzPpKvE+++wD+A9mOBzm3XffBXBOnHjqoB72//3vf26jDBgwAID//ve/1frbiVSfCgoK+OCDD8q9Nm/ePM4880zAX1881Vzq03HHHee8qBp0Lc4++2ymT59encutRKpV4gsvvBCAZ555xr2mw+bII48E/Hg6+LHZtm3bAjBy5Ej3vkWLFgFw//3388UXXwBVmwfpVIkbNWrkDuRt27YBngCSk6l58+aAJ4AAt57aYCqxYdQDkq4SX3/99YCvKi5btqxKyar3TZs2DfDUjBUrVgDw1ltvJfty98i6detcyEnq+6mnnlott72k7yuvvMIxxxwD+Oq91isnWhCQOhjL5s2bgfIhLElWhe/knPn2229ZunQp4Eul/Px8d58zNZZdUlLitD2t6Ve/+hWtWrUC/OvWEPNkYBLWMAJE0iSsTtejjjoK8CXKOeecU+UJKofGcccd517TMOh49mGqWLNmDXfeeSfgO84ikUiN85xPPvlkwJeshYWFgO+oCALvv/8+4IdkmjVr5nwOcsC89tprdOnSBYDevXsDcPzxxwOwYsUKxo8fD8DixYvd75IGk6lkZWWx7777Ariw1JgxY9y9XLlyJUBSnYcmYQ0jQCTNS6yhzT//+c8BP79y+vTpfPvtt0B5W0WnlKobDj/8cMCzG1RnWFM3eaI9irJblJa2detWZ3dVx5bNyclx9p/sOWkgH330UXUuIS7pSpw4++yzAfj3v//togEKb5WWljr7tCJjx451iTI1Td1Lp5c4JyfHPZfKa2/fvr37/oEHHghQ67TLWPa0zqSpxFJvlHebnZ3t/cFIxG3AWAfFEUccAfgbVUyePDlj4nNakw6Vpk2buuwd5TbrAYxV37X2e+65x71/1qxZQLCcTRVRGOrxxx/n6quvBvyYK/gHrD4TxdafeOKJWufYppNIJOKKHNq1a+deX7hwIZCYjfpDmEpsGAEiZbnE8QqblTWSl5fn3PxKMhAtW7Z01RA1JdnqUzgcdmrQRRddBPhOpZycHHfdBxxwAOCtU9JZJ3UiHBTpbnNaUFDgEiBk2oCvZai8TiG+hQsX1jp0k06VuF27ds50kUkTjUbJz88HfKdTIrDECcOoB6Qsl1gnanFxsZO2sgPGjx9fSbKmIghdV6LRqKsquuaaawA/LBVb2C5JM336dIYPHw7UvEolk9m1a1c5ySrmzZsHwLBhwwC/YUGmJkbsCTlQr7vuOhfWEVu2bHF1vqmgzhs2KyvLOVKUX/lDDgU9zH369AGgZ8+eld6jzJlMjs1Fo1Fat24NUOlGFhcXO2eT1nvssce6w6o+8cADD1R6rbCw0PU42r59e6ovKaGoOP+qq66qdDDNmDEjpU5RU4kNI0DUWsIqJnnfffdx/vnnA/Dwww8D8MgjjziVT+qgTqbs7GwnkSVhJYliUTFwpiPtQHHm559/HvDKxg477DAA7r77bsAzAaZMmQLgukQqLzn2lNZnlZWVVan1TCYhZ5ocbeCvo1evXoGXrAo//vOf/yz3f/Dvx4033mgS1jCM+NRawqpqYciQIc7RcvnllwNeNcfLL78M+LaoJG1OTo6TzsrDjeewqG1taKpRd8cHH3wQ8IPnpaWlLlFAzpcPP/zQ5UhL0g4ZMgTwsqZ0UsvOzc7OdsklmSRhc3NzAXj77beB8vdPOePKkQ4yRx99NOAXpscyYcIEIDXJErGYhDWMAFFrCasTf+fOne6EVcXG4MGDWbJkCeDbaLHvV19iBZ/jXlicFqiZyI4dO4Cqg+ZKoLj44ot54YUXAFxdrLSSf//7365uNrbVayZ5yeXtVvO1eGmIyrFNZ2VVorjllluA+BqE7luqqfWu0IPar18/l8nSpEkTwBu9oQT5xx57DIBXX30V8DbweeedB/iNpWNjlrrx6rhXH9CaFixYwKRJkwAYOnQo4Le9mTRpUiUH0+7duzMmjxr85Paf/OQn5V6PRqNOddcmbtasmcuvFhX7FGcqeh51qIqysjK6d+/u/p0OTCU2jABRZ73z008/daeqVIgbb7yRNm3aAL5TZuTIkYAnmVWKFesmFyrPeuedd+p6aRmJGnZJIikcFIlEKg2PyiRJFAqFuPXWWwE/DKfrDYVCTlVUqGfAgAGu2Z40Br0n08M9MtWkMYqvv/467Z0tTcIaRoBIiGdHJ+1tt90GwMSJE10TNbXzVKiiUaNGcdPzdAqr9Uamn8I1QU6Ltm3bctpppwHl7VqAtWvXZqRkFeFwmG7dugH+epSCGg6HneTR11WrVjltqaI2kelo0oTWqWdzzJgxafcpJMUVu2zZMmecK2anrKZhw4a5WKS8xatXr3ZJ8XXpvJBJhEIhpzoqF3Xs2LGu4EEOGcWbd+zYkVEe4YqEw2Hn8deBIk9+cXGx25x//OMfAc+kCcoGrch7770H+P2vVfo5ceLEtF2TMJXYMAJEysdNppJ0FDtLjWrcuLEbR/GrX/0K8KSUPu/XX38dgDPOOAOomwmQqgJ2lRIqJKVspgULFrjSOYX7kqU6pnvcZKqwAnbDqAeYhCU5rXCOPvpo5s+fD/i2XjQadcXO6pZYH1rEpBKTsIZhBAaTsCRnndnZ2a4SRxPuxo4d69I4E9nm0yRsZerrOm3DsvesE/aetdbXdZpKbBgBokoJaxhGZmES1jAChG1YwwgQtmENI0DYhjWMAGEb1jAChG1YwwgQ/w8mjR/dXzEhOAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x288 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "generate_and_save_images(model, 0, random_vector_for_generation)\n",
    "\n",
    "for epoch in range(1, epochs + 1):\n",
    "  start_time = time.time()\n",
    "  for train_x in train_dataset:\n",
    "    compute_apply_gradients(model, train_x, optimizer)\n",
    "  end_time = time.time()\n",
    "\n",
    "  if epoch % 1 == 0:\n",
    "    loss = tf.keras.metrics.Mean()\n",
    "    for test_x in test_dataset:\n",
    "      loss(compute_loss(model, test_x))\n",
    "    elbo = -loss.result()\n",
    "    display.clear_output(wait=False)\n",
    "    print('Epoch: {}, Test set ELBO: {}, '\n",
    "          'time elapse for current epoch {}'.format(epoch,\n",
    "                                                    elbo,\n",
    "                                                    end_time - start_time))\n",
    "    generate_and_save_images(\n",
    "        model, epoch, random_vector_for_generation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P4M_vIbUi7c0"
   },
   "source": [
    "### 使用 epoch 编号显示图片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WfO5wCdclHGL"
   },
   "outputs": [],
   "source": [
    "def display_image(epoch_no):\n",
    "  return PIL.Image.open('image_at_epoch_{:04d}.png'.format(epoch_no))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5x3q9_Oe5q0A"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 287.5, 287.5, -0.5)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dd3SU1fawnymZSaMFA0YgRECDWECUohRF4xWlqGBBxN4brqv8xIICLu9Vr10U7F68KlZE7AiKiiBSFCWCoig9QEKCKaTMzPv98X7nZAIhmSRTzsT9rMVKMsy8s+eds88+e5999nZYloUgCObhjLUAgiDUjiinIBiKKKcgGIoopyAYiiinIBiKu57/l1CuIEQeR20PiuUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEMR5RQEQxHlFARDEeUUBEOpr+J7VHE4ai18HTPq610q8jaN5iZvuBHLKQiGIsopCIYiyikIhiLKKQiGYlRASKgfl8sFgN/vj6kMlmURCARiJkMoqHuVkpJC3759OeKIIwBo27Ytubm5ALz++usxk68+xHIKgqE46gkPRzV2XFvoXM1+Q4YMYcKECXTo0AGAK664AoBly5btdwZPTk4G4JFHHmHIkCEAvPnmm9xzzz34fD6g7vB4pEL9bdq0AeCVV14hLS2N0aNHA5CXlweA1+slOzsbgFNOOYVLL72Ugw8+GICcnBwAvvnmm6jJq5g3b56WCeCXX34BoFevXpSXlzf4eo2R1+PxAJCfnw9AixYt9vt6tbqorKxk69atrFy5EoDRo0fjdNp2Scndrl07iouLmyRvE5DmuYIQTxhvOTt37gzAihUrSEtL2+c55eXl/PTTTwBs3LiRoqIiDjjgAADOOOOMWt9n9erVjBkzBoA1a9bs1/JGyhJ5vV4AcnNz6dKlC0uWLAHgxhtvBGwLWlRUpGVo0aIFM2fOBOCcc84BoKSkJGryKpRlb9++PYFAgLKyMgD69OnD2rVrG3y9hsirfh8/fjwAjz32WK2v8fl8LF++HIABAwYA6O/X7bZDLAsWLGDw4ME1Xte1a1fWr1/fJHmbQK1fjPEBIfWleDyeWgdXYmIiffr0AdA/66NTp06ceOKJAPz555+UlZVFNftDLbdatGiBZVnMmTMHgO+//x7YdxCUl5fz1Vdf6d9jRXAQqqioiMrKSgA9GUYSpVjqewvGsiwuuOACAN544439TrZqUuzbt+8+/1dYWBgmScOH8cq5fft2AHbu3Fmnf9EQSktLad++PQAZGRls2bJFD/poKulXX33F7NmzeeONN+p9byWf8pVjgbKUPp+PRYsWkZaWBtgrkUhTVVUFwNNPPw3AyJEjAdufPProo7X/W9c9POusswB7Qt/7urGc9PaH+JyCYCjGW041o9122208//zztGzZEqi2IHl5eTpyl5aWRocOHXSEtzaqqqp47733WLNmjf47NTVVL9GisX+o3mPq1Kl1+rwKl8ulI42xZNeuXYAdB8jMzNRL7d27d0dNhi+++AKwI/AADz30kF5d1ceMGTP2eWzBggUAVFRUhEnC8GG8cqplyieffMKUKVO4/vrrAfj4448B+Oijj/RzTznlFMaOHUvbtm2Baj8lmPz8fEpKSkhISACgVatWtGjRQgdgoqGcqampAGzdujXk91u0aFEkRQqJH3/8EbC3TrKysrjjjjuA6LoCahK9/fbbAUJOhOjQoYO+78Eo5TTtBAzIslYQjMV4y6koLi7mmWeeYceOHQC88MILAFx33XX6OT6fj5KSEjZt2gSgExaKi4tZt24dAOvXr+eQQw7RofR169aRnp7OrbfeClRbh3CjLPXMmTN1csSkSZPYvXt3vdYzlkGgYJSV8ng8VFRU8Mknn8RMllDuiUo0mDlzJuPGjavxf+qeZ2RkAHDwwQezceNGbZlNQCynIBhK3FhOh8PBoEGDeO6554DqPatgPB4PaWlpOsSvfKGSkhLefvttwLacycnJDB06FLCta69evbjooosAmDBhQkTkv+qqqwA4/vjjdQDl9ttvp6qqSm/uq431HTt2GJlUfsghh+jfA4FA1CsDhEJiYiKtWrUC4NdffwXQQURFZWUln3/+OVDty7vdbhISEoyynHGjnF27dmXWrFkkJSWF/Brl5Ofl5bF06VLAVs6KigodnTv33HNJTk7WCh0pOnbsCMCdd96p9wX9fj+DBg1i4sSJAFx55ZUAdOnSJaoR0FBRS0DLsnA4HPr+xlpJXS4XAwcOBGD27Nk6d3nvII9aCj/xxBM89dRTAHTr1g2Afv360bNnT+bOnQvAnj17oiJ7XRivnOoGX3XVVTVmQBVdveOOO3Ta1dChQ7n88st1soKyPh9++CErVqwA7JC5w+HQieM9evRg+PDhLFy4MCLyqw3vadOmAbBt27Yag3n16tU888wzQLUf1KdPH+bPnx8ReZqCGtyBQEBvQQH1JoxHmrS0NGbNmqV/3x9//PEHAG+99Zae/E4++WTAzjzKz8/XWzUmKKf4nIJgKMZbThVx6927Nz6fT6eQ7e1XAnz66ad6Yx+qjxe9+OKLNXwJh8Ohl8fZ2dlUVlZGzFKpKO3WrVv3+5y9/cuJEycaaTnVcrxdu3a43W5OP/10AJ1+GCtOO+00DjzwwDqfY1mWHjuHH344Rx11FABnnnkmYCfzJycnG5WMYLxyqmXtmjVr6NGjh14e7s/PKS0t1RlD6gs7/PDDtZJYlkVGRgbXXHMNAAMHDmT79u16iyZcqASIhnzZSknbtWsXVlnCxbvvvgvYE86gQYN0rmosldPlcjFlypR6kwgqKiq02zBixAh69OgB2P69us727duNUk5Z1gqCoRhvOVUQ4uabb2bp0qV6S0KdgVy8eLF+TuvWrXnmmWc47LDDgOoqCp9++mmNawafRVyyZAlXXnll2Df6Vb0aFc4PJUT/4YcfAnD++eeHVZZw8cEHHwBw5JFHcuqpp+pIp8PhiHrEVgXaVqxYoatEgG0hH374YcCOyoJ9NM/tdusgUEZGhs7NVZ+hrKyM2267TSynIAj1Y7zlVFRVVfHKK6/w/vvvA/Dss88CcPrpp7Nt2zYAhg8fzvHHH1/nqRSAzZs38+STTwLw/PPP622ZcBC89QMwffp0oO4zjypR/9FHHwXMCOPXhgrOpaenk5CQoO97LFAHprt27QpUxyB++OEHPUbUfb3xxhsZOHCgtrZer1cHC1UVjc2bN7Nt2zY9dkxImYwb5VSopcnVV18NQFZWli7e1aVLFx342RsVbFm5ciV33323joaqw7bhpnXr1gA6Z/fiiy8GqjfwwR4kffv2ZcSIEYB9hEw9x0RUgG306NG4XC69DI+FvGqiKC4uxuFw6L+7dOmi77UKrClFVs9Zvnw5mzdvBtCZY4WFhfz5559Rkz8UjK8hFApqtuvVqxfz58/XiqEIBALax5g6dSplZWUhpcc1pSbPMcccA1RPIsoaBgIBOnXqBNjJ1mVlZdx///1Atc/ZWCJdQ0j5+cceeyzr16/nyCOPBELzp2ujKfKqFL3p06eTk5Ojs4J8Pp+2hqom7Y8//siqVav0xO7z+fR7q7GjLGlwxb6GytsEpPqeIMQTzcJyBtOuXTuef/55AL1J/sorr2gfsCGzfFNmdjUjqxS3e++9F4ANGzboGfzHH38kNzeX0tLSkN4vkvKGgtoLDgQCdO3aVcvdWMIhb0JCAi6XS/uTPp9Pr1LCfXA+2paz2Sln8HVGjRoF2MWQG5P/2dz6RzZVXpXbXF5eHpbTG83t/jYBWdYKQjzRLC2nQkXnGns2srnN7CJv05Ci0mHExAPLghAqRllOQfibIj6nIMQTopyCYCiinIJgKKKcgmAoopyCYCiinIJgKEbtc8bbprPI2zSam7zhRiynIBiKKKcgGIoopyAYiiinIBiKUQGhWKFOr4AkywvmIJZTEAwlbi2nCrO3bt2a3r17AzB27Fg6d+6sS4Oo/iSvvvoqy5cvB+yKd0lJSbqERUlJCVBdgmPPnj1RD5mb0kqvPpScY8aMYceOHXz33XdAzdIvGRkZ+Hw+tmzZAkTuM6nVjtfrpUuXLrr3yXHHHaerMapCb/n5+cyYMUN3lsvLy6vRgwfsNh4//fSTUa0XxXIKgqEYdZ6zIZvOjz/+OADXXHONbhqkPovyG7dv3w7YDXNVfdKPPvqIgoKCGr0mLcvSs39wUahIb5J7vV4uueQSLr/8cgAefPBBAL744osaZRyDqUumSMuragIvWbIEh8Ohi5YlJiaSnZ0NwMcff8zKlStDqgfcFHlVwejp06dzxhln1CgkrqxicCzBsizKy8sB25KqgmBqdTV+/HiWLFlSp9xSCSFEVOu2hIQEPYC3bt1Kbm4uP/zwA1Bd+c7pdLJx40YACgoKKC0t1TfasiycTmdUA0GqUNaCBQvo3bu3HkSqiv3y5ct1j5WZM2fy22+/6Up3sezloe6Rx+OhoKCAAw44AIDLLrtMy/fEE09ErFB3MOo+HHroobjdbj2pbt68mdzcXMBuQgzomrYFBQX6c6gWkDfddBNgt583LRgYt8qpqqSPHz9eNyqaP38+ZWVlerCrRjYHHXSQbm6Um5tLeXm5/iLUz+BZOpK+X0ZGBt9++y0AnTp1qtEEKCUlBbD9IGUJEhMTqaqqMqI9gLpHBQUFbNu2jYsuugiAzMxMzj33XAB27doVFVlUrODaa69l1qxZeqUxefJkFixYAKAtZZs2bRg8eLCuxjh48GC9Uvr6668BM6P04nMKgqHEreVUJfdvvvlmbXnKy8txu926TYCazcvLy/Us+vXXX5Ofn6//9vv9NSxlpKymihyuWrWK9PR0/V6BQEBbxb/++guwVwDKp1b+mwmRXOVzdunSha5du+rl+axZs3SrhmizdOlSLrvsMk455RTAjjOo71bd16KiIjweD6eddhpg+/qfffYZYKbFVMStcqrBWlxcjNfrBezl66233soll1wCVPdw9Pv97Ny5E7Arru/Zs6fGsjYaA//6668HIC0tTftkGzZsYNGiRbpB0Ny5cwH47LPP9FaEKYoJ1ctu9bu6p7fddlvMBrllWSxcuJC1a9cCdtMi1Rv1t99+A+zJY9iwYXpZXlJSwoQJE2Iib0OQZa0gGErcWk6FZVl6Rv/oo4/Izs7WwRQ1m1dVVWlLtHv3biorK6Nujc466ywt07Rp0wCYNm0aFRUV9OvXT8sG1YEMiE3X6P2RnJwM2NHviooKnnvuOaB6OR4rLMvSvUJzc3M555xzALjllluA6m0XxZw5c1i/fn10hWwEca+cYIfyAQ477LAaUVe1h1VWVkZGRgZgNxiKxWBX+4AVFRXMmDEDgG3btpGamqonF7UdUV5erv0lh8NhjIIecsghgK2cv/32G2+++SZgVmbThg0btDLurZRqGX7XXXcZ7Wsq4l45HQ4HY8eOBWyrVFhYyG233QbAsmXLALt79dlnnw3Aa6+9xqJFi2Ij7P+nc+fOgB2wOProo3Uvz6eeegqwlVQNnr33YWOpCKoZbSAQoKCgQK9GTMLv9+uOcnuzcuVKwIyu1aEgPqcgGErcW063283xxx8P2MvB2izL+PHjddJzdnZ2TCynCuMvWLBAt7wPBAIEAgFtgR577DHAjuiqbQu3201iYqL291S79Pz8fB31jZY1VffZ6XTy8ssvRyUTqKG0adOGFi1a1HisoqKCwsJCvWI5//zzefjhhwGzt1LEcgqCocRt4rvH49E/y8rKgP3PgpMnT2by5MmA7Tfl5+eH9B6RSCT3er3cfPPNAOTk5NCzZ0/difnuu+8GoLCwUEdsu3fvTqdOnbTFnD59OlB7jm2kE99VfvLq1asZPXq0lruxRELe1q1bU1hYCKB/zp49m0GDBpGVlQXYfqnaW25IpFk6W4f4nEMPPRSwl3cqn3Pvz6Keu3PnTp0ErxITQiHSg93pdHLqqafyzDPPANWt6VesWKEHlt/vx+fz6UhjXR2lIy2vus8TJ07U2yhNIRLyOhwOvvrqKwBuvPFGwJZ76NCh/Oc//wHsgwf9+/cH0GdSwyFvE5AuY4IQT8RdQEhZzAceeACASZMmUVRUBNQ8iwnw3//+F7CDBBdccEH0hAyRQCDAvHnzeP/99wE7FRHso27qLOqOHTvqtJbRQAWj1Orj+++/j6U4dWJZFpdeeimATjSwLIuZM2dy3HHHAXDJJZdw4YUXAg2znNEmrpTT4/HwxhtvAGifoVu3bvz555+AveTz+Xw611Ydadq6dSuvv/569AUOAb/frz/TuHHjANi0aROrVq0C6l7GRgOn06lzVdVRPHXowFRU9Ds4BlFZWal9ZoBevXpFXa6GElfKmZaWRvfu3YHqVLcRI0box0477TR69OihM0PUl5OZmRkDaUNHZS+pA+QPPPCAMbVsPB6PXnXMmTMHiO2B7/pwuVz6+/7ll1/04wkJCVx33XX6b7XaMhnxOQXBUOIqWutwOBg+fDgA7733Xq2v8/v9vPzyy0B1zm1jiUajnf79++ukCBVlPOmkkxp1rXDKq1L1PvroI+3nq/Ob4SIS9zctLY1rr70WqF66Dh06VPvLYFdy6NChA9CwVYBEawVBAOLMcgY/57zzzgPs2Vxt2H/77bf89ttvYUvJirTlTElJobCwUKfqKV+5sXV4wiWv0+nUdZlycnJ01LNr166Nkmt/ROL+pqam6tMyQ4cOrXEd5Wd27dq1UfdYqu/Vg7pBpkZfG8Kdd96pFROiVxyrPlwul5YrPz+fnj17xlii0CkrK9MTi9o6KSoq4r777uO1114DqouDmY4sawXBUOJuWRtNIr2snTx5MldffbUuKv3xxx836XrhlFcV6lYnZyKBdLbWNI/c2mgS6cHjdruxLGufzKbG0twGe7zJ2wTMV05B+JsiWymCEE+IcgqCoYhyCoKhiHIKgqGIcgqCoYhyCoKhGJW+F2/7WiJv02hu8oYbsZyCYCiinIJgKKKcgmAoopyCYChGBYQEM1FV9zp27AjYRbr315dGCB+inFHG4XDowZ6SkkLHjh058sgjAcjLywNgyZIlMS+JqfB6vYwaNQqAZ599FrBb6Z1//vm6/2m8oO67x+PRE0twDaHg78btdutWgeE6NdRQ4kY53W43LperyWUZVXhe/bQsK6IWQFUUGDZsGAAXXnih7tbVunXrGs181c9169bpQmYbNmyImGx1oc5z3n777brfqaoH3LNnT/r168cHH3wAUGe3seD7HCucTiedOnViwoQJAPTu3VtPhKo/zZ49eygtLdXKWVxcHDOlVBivnKpq2hdffEFmZiZ9+vQBqFEguD4yMzN1hbs9e/YwbNgwNm3aBIS3aLPT6dSDWrWYU01727dvv8/zA4EAVVVV+zRz7d69O7NmzQJg4MCBUW9T53A4uOmmmwB78O6tYNu3b6ekpGS/ciUkJOgGtqpgdqjNo0KRDWoeBt/fBOtyuQBo0aIFJ5xwAgcddBBgVxZUtYLV9YqLiyktLdUK6fP5Yt4eUAJCgmAoRltOh8NB7969ATj66KPx+Xycc845QLVlqgtVNT243uquXbvYuHFjWC2m6lyWlZXFyJEjAbjmmmtqPKe0tBSAZcuW6f4cGzZsoKSkhGOPPRaAs846C4AOHTpw2GGHAfbsH+0ZvF+/fjz44INAzSwd9RluueUWFi1aVOuyz+l0MnfuXF35TpVeCZflVCh3weFw6F4uYPuHqjWhWm6XlZWxcOFCUlJSADj88MNZs2YNAD///DNgZhNd45VTKdiaNWt48cUXdbu8ukhKSgJqKqUqn5mRkRFWxXQ4HLpvy5gxY/RyTg0M1VH79NNPr/G4eq3L5WLlypUAOvBiWZZWisTExKh2kPZ6vSxYsKDW1LnRo0cDdvHr/fn+ffv25dRTT9V/K7fE4XCExe9U11AK6HQ6SUpK4oADDgBsP3716tVA9XfucDjo3r07d9xxh37OP//5T8BMpVQYrZyBQEA3zTnmmGNC9gNU/VdFeXm5VtRwD3TLsnTvzLfeekt3B1M+o2rOWpvcqn6Q6u2hZvZAIKAHvwpQRItp06bVsERQrQhff/01YEc491a0bt26AbBw4cIaiq2aHIc7IKSuFwgE8Pv92gd1u906cKUm4YMOOojnnntObwX9/vvvfPbZZ2GVJxKIzykIhmK05YRqi9OQpegJJ5wAVM+uGRkZEV0aqrb3a9asITc3t8GvV9Fa1bquZcuWusp6tDp6KR/ukksuqfG43+9nzJgx+newl4kOh0NbqyOOOIIPP/wQqN5uUUS6XaBlWQQCAdLS0gAYP3482dnZALo1ZHZ2Nh07dtTj4R//+Mc+EXITMV45G8OQIUMA+OSTT4DIt3vbe5+yoa9dsmQJYLcwBLjiiiv0/ma0lFO1WlBKqj7L//3f/zFv3jygeqJMSkoiPT1d19sdPXr0Pq6EIhr+8l9//aXb/f3+++/k5OQA1Z/J7/djWZYeB7HaO24oRpXGDNf5PZW5MnDgQKC6w3FDifZ5Q2V1Zs+era3xeeedF3LQoinyzpgxA6iOMqvo6hFHHKF/V69PS0tj1KhRTJkyBbA7hysrqnxk5afu7b+GS966UMGhV199FbAtZ7t27fRe96mnntroiTRCNI9eKfWRmppK69atAdi2bVuMpWkYamAPGjQo6geN1XaOQilk8PJPDU6/38/atWtZtmwZYLfaUwrh8XhwOBw8/vjj0RC7VpTsqrP5lClTuOyyy/QkFy85wRIQEgRDaXaWs3///trqxDo3sqEoK5WYmKhTz6KVhKCWfMqCqv3JkSNH6kQC1QWtrKyM3Nxcpk+fDsDgwYPJysoC4OSTT8bpdDJt2rSIy1wfaotr0aJFXHXVVbRp0ybGEjWMZqWcbreb1157TQc14k051XLL5XJFfX8zOEhiWZbeF77rrrt0h/Bx48YBsGPHDkpLS/njjz8AO3dVbfwHAgHcbrdOHjGBnJwcnE6nXnrHC81COVUwYv78+aSnp+tEgHhTTvU5gv1Nt9sdlYinOiljWRalpaV6GyI1NVX353z77bcBeO6559i1a5fe1N+yZYtWRqfTaYyvr9Iq1daKKcfwQkV8TkEwlLi3nImJiTz66KOA7ftYlsXJJ58cY6kah0p8D85DjdZmufJrS0tL+fjjj7WVzMzM1FHPXr16AXDfffexY8cOfv31VwAWL17MUUcdBdgWeP78+UaUtezXrx8AhxxyCGAvx+OJuFVOFTDJzs7WCdlgn4JoTJaOCbz00kv6d7UdEK2lufIZu3TpwtSpU/Wy1ul0Mn/+fAA+//xzYN+l76+//qq3r8rKyli9enVUk/WDcTqdOkd5xIgRgH0PfT4fjzzySExkaixxq5xq0O7evVtv3vt8PsaOHRtLsfYhlEoASn7107Isbr75ZiB6pyaKi4sB21Kefvrp/Pjjj4CtrCoxf/PmzYCdDpmamsqgQYMAu7aQOsjsdDq1RY0Wbreb9PR0wD79o/xf5cPv3LmTwsJCPbnEC+JzCoKhxK3lVLhcLh2V27Rpkz6iZQpqiXXEEUcAsGrVKr3kS01NZfjw4fucUd2wYYPOCw7XOcj6UOl2iYmJTJw4UZ+JTUhI0H6vOh/p9Xrxer16VXDwwQfrdMO1a9dSVFSkVwGRyg1W752cnMy///1vTjnlFMC+33PmzAGqS8Ns3bqVDz/8UMsfL8S9clqWpQfApEmTjEvNUhPHwoULAVteFdJPTk7WSy+oPt0yZswYCgsLgeoTIJFe3q5YsQKwKwB26NChxhJbyau2SH755RdWrlyp//b7/Tqf+a+//mLjxo0RDwgFpxK+9dZb2i8eMmSIToJXp3y2b9+ukyziibhPfO/WrZu2MieddFKDCn/VRzgTs9XgOeGEE2ooZDAqurhixYpGBYKaIq9KekhOTiYtLU0nclRUVFBSUgJU+6WqqNb+3kP9g7oDWuG6v06nU8t/wAEH6MoUKkiVl5dHYWGhPpVSVVXVqJNEkvjeQIqKivSRpvLy8qgtAxuKOsbk9Xp1BcFzzz2XVq1a6UCFqi0UC5RlLikp0crYGCJdarQ2gieLnTt36ki3mmCcTid+v19/xlAnj1gjASFBMJS4X9Z6vV5dPGv9+vWsWbMmbGlaza1/5N9V3rquI8vaCOL3+1m3bh1g+0vJyck6Gmri8laIPvE6DuLecqrykgpVkiIciCWKLM1N3iZQ6wcVn1MQDMUoyykIf1PEcgpCPCHKKQiGIsopCIYiyikIhiLKKQiGIsopCIZiVIZQvG06i7xNo7nJG27EcgqCoYhyCoKhGLWsFYRo4PF4dIUK08raBCOWUxAMpVlaTnVKRTXT2bJli+7VuG3bNiorK+P2GJEJuFwuPB6PLvwVqxq1DeHAAw/kySefBGDYsGFa9hYtWsRSrDppdsqZkpKii02pZjyWZTFp0iTALp78yCOP8O677wJ2jZxoKOrex9rikUsvvRSAJ598Eo/Hoz/H4sWLGTlyJECTSpxEisTERC644AJd2xaqJ5RQ6grHCqNOpTQ1dJ6YmEhxcfE+BbR8Pp+ujuB0OgkEArzwwgsA3HPPPRQXF+uBFnwetCmh/gEDBgDwzjvvANVlGi3L0u/19ttvc+mll4atZGOktiZU8ay9JxVVkycQCOjCaieeeCKbNm0K6bqR3kpRcvfq1YuWLVvq2kIzZszQFfquuOKKkK8X7fOczUI51euKioq0tQR01fQvv/ySzp07A3ZLu+7du+u+GaNGjeLnn3/WyhtcgrIpgye4DmyoPP300wBMnTpV95ZsiJWN1GDfuyU92DWCVdvAQw89lFatWgGQm5tLTk6OLu0ZC3kVbdu2Bez6wLt27SItLQ2ABQsW8NBDDwHV9zwU5LC1IAhAM/E577//fqDax1Q9PR577DHAnoFV/45+/fqRlZWlLefvv/8eEb9zb4upLGBBQYG2CIFAgPT0dL38Upbpmmuu0b5b586ddUfpWNC2bVuuvvrqGo99+eWXXH755fF1qaYAAAhoSURBVFrG9u3bc8455wB2J+x7772XqVOnArHt7KXu6+bNm3E4HIwaNQqwa9uuX78+ZnKFjKozup9/UQV7Gd2gf0lJSZbf77f8fr9lWZa1dOlSy+VyWS6Xq8bz0tPTrfT0dGvnzp1WcXGxNWDAAGvAgAF1Xrsp8u5Ny5YtrZYtW1pOp3Of56amplqpqanWunXrrHXr1tV43ciRI0O+F5G4v+edd55+fUFBgVVQUGBlZmbqe+xyuSyPx2NlZWVZWVlZ1owZM6xvv/3Wmj59ujV9+vRav4tIylvbfXU4HFarVq2svLw8Ky8vzyovL7e6detmdevWrUHXiyC16l/cW84JEyZoS7R27VqOP/74ffw0h8PB7NmzAXvW/OOPP1iyZElE5VKherXJXddmt7JA/fv3B2xroz7Tzz//HEkx60WtQgDdvUsFsFQE2u12a38uLy+P3NxcXW1dPScWEepg+W644QbatWunZVHV301GfE5BMJS4tZxqVuzdu7f2a84888xaZ+i2bdvqrQ2wezhGujHQfffdB1T7waHsp5100kn6uapJUKx9oxUrVujIq9onHDVqFJ9//jnZ2dkADBw4kMGDBwPwxx9/8OKLL7Jq1SqAsBX4bgxt2rQB7FYY99xzj/4O/vzzz5j68aESt1spycnJgL1PpTaU586dS15enlZQdb0ffvhBt0X3+Xx4PJ6QAkD1PacuedVeq2rV/tNPPwF1t8RTk0xaWhq9e/cG0E1sQ6Ep8tbFGWecAaBb0RcUFOD3+/XSNZgZM2Zw5513htT6L1LyKo499lgA3n33XTp27Kgfz8zMDHkvNphQxkwjaV4V35VP98477+iGNW63G4/Ho32inj17AmjFBHjzzTejkg2i5Pvhhx8AtE9WXFysB66y3kp+tVf42WefxdzXDGbu3LlAtbIEJ1SAPeGoaPjLL78csZ6cDeXiiy8GoEOHDgAsX74coFGKGQvE5xQEQ4nbZW0wyv9Uydhqf0ula6Wmpurnpqen68frI5zLrqysLAAuuOAC7Vt6PB7y8/Pp1KkTUN2c9uKLL26UTxTpZWJwC73gvxcvXsyECRMA2zqFGpmNtLwFBQWAvWqxLIsuXboAts/ZGKK9rG0Wyhn8epfLRWZmJmAnGCjUgElMTNRLzvoI5+A54YQTAHtZrdLK1CSiBrkKuISS+lYbkR7se19fdeu+6KKL9GEDE9INVTxCbVE5HA52796tXYvGBgPF59wPLpeLVq1aUVpaCuw/sOJ0Ohk0aNA+j6s9ulAVM9yobssqggj2fmFCQoJW0uCTK6azbds2hg0bBkBZWVmMpanJkCFDgJrK/f777xt58qQujFdOFfV85JFHOPvss5k+fTpQvVURCAT0l5CQkECrVq1qVc4FCxZESeLa6du3L2AfVZozZw5gJ7gfeeSRPPjggwC89dZbgL0lpJIW1IBSn3F/J0QiiVp2KyzLol+/fsYpJdiuwksvvVTjMb/fzx133BF3yikBIUEwlf3l9VmG5Naq/MgdO3ZYfr/f2rBhg7VhwwbrwAMPtA488EArKSnJ8nq9ltfrtVq0aGEddNBBOq8zmHHjxlnjxo0Lay5lQ66VlJRkJSUlWVlZWfvkm6q838rKSquystKaN2+elZaWZqWlpVkej8dKSEiwEhMTrcTERCshIcFKSEiIWq6q1+u1tmzZYm3ZskVfJy8vz3K73U3OfY2EvAMGDLACgYAVCAT0dV544YUmyxqKvE0gPnNr1fKtoqICh8OhgyZqU/mvv/6q8ZyUlBTt+Aez9wHsaKPOd9YWKVTRY7U/279/f6688koA/ve//1FSUqJ95WjtIarl84wZM/bZ13z33XcjnmHVWO666y7tAqikDnUv4w3jlVMN6pycHBYvXqxrvqiN8eeff5558+YBtqKOHj1al8xQWJalDwabiBr0b7zxBgBjx45l6NCh+jErqHqCFSW/SUW8zz//fP2Ymjzat29Pq1atKC4u1v8XXBUhFqjJRB0eAOjRo0dMZWoq4nMKgqEYbzkVv/zyC+3bt+euu+4C4O677wZg0qRJTJw4EbCtbFJSEh6Pp8ZrCwoKWLp0aXQFbgSqrk15ebnOF3a73fh8vqhWuHM4HPr+JiQk6CW1WiZ26tSJoUOH8t133wG266H+L1YRXOXKBFfTi4fk9rqIG+UEe49y8uTJAPrkfbdu3fT+YEpKSq1V7nJycowM+yuUj3TaaacB9tJ10aJFAGzfvp2qqqqoLs2cTieHHXaYlk35uWqw79q1i02bNukMnODJJFZ0794dsOWNtgsQKeJKOYNR/oTX69X7mhdddBEDBgwgJSUFQPttDTnZEU0cDgcJCQl603zgwIGAnRyvfOo9e/ZEPXHC6XTqfdZAIKCDaUoZ//Wvf7F06dKYK2Qwy5YtA+Cbb74hNzc3xtKEh2aVvhdu6pt5m1otMDU1lXvvvZcbbrihxuMLFy5k+PDhQMOWieGUV1UrHDt2rM75ffPNNwF7wgiHVYrU/Y0UEbTEUn1PEOIJsZx1EKmZXfnFffr04euvv9bLRrXd07t3byNPpYSb5iZvE2j+p1LCTaQHT0JCAmPGjGHEiBEAXHjhhUDjEw2a22CPN3mbgChnQ2lug0fkbRricwqCABhmOQXhb4pYTkGIJ0Q5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQRDkFwVBEOQXBUEQ5BcFQ6utsbVYnGUH4GyGWUxAMRZRTEAxFlFMQDEWUUxAMRZRTEAxFlFMQDOX/AcAxkkaLKOArAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(display_image(epochs))\n",
    "plt.axis('off')# 显示图片"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NywiH3nL8guF"
   },
   "source": [
    "### 生成所有保存图片的 GIF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IGKQgENQ8lEI"
   },
   "outputs": [],
   "source": [
    "anim_file = 'cvae.gif'\n",
    "\n",
    "with imageio.get_writer(anim_file, mode='I') as writer:\n",
    "  filenames = glob.glob('image*.png')\n",
    "  filenames = sorted(filenames)\n",
    "  last = -1\n",
    "  for i,filename in enumerate(filenames):\n",
    "    frame = 2*(i**0.5)\n",
    "    if round(frame) > round(last):\n",
    "      last = frame\n",
    "    else:\n",
    "      continue\n",
    "    image = imageio.imread(filename)\n",
    "    writer.append_data(image)\n",
    "  image = imageio.imread(filename)\n",
    "  writer.append_data(image)\n",
    "\n",
    "import IPython\n",
    "if IPython.version_info >= (6,2,0,''):\n",
    "  display.Image(filename=anim_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yQXO_dlXkKsT"
   },
   "source": [
    "如果您正使用 Colab，您可以使用以下代码下载动画。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4fSJS3m5HLFM"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  from google.colab import files\n",
    "except ImportError:\n",
    "   pass\n",
    "else:\n",
    "  files.download(anim_file)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "cvae.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
